Papers with computation time
Computationally Efficient Wasserstein Loss for Structured Labels (2021.eacl-srw)
Copied to clipboard
| Challenge: | Existing approaches to estimate the probability distribution of labels are based on tree-Wasserstein distance. |
| Approach: | They propose a tree-Wasserstein distance regularized LDL algorithm for hierarchical text classification tasks. |
| Outcome: | The proposed method performs well on synthetic and real-world datasets and compares favorably with the Sinkhorn algorithm in terms of computation time and memory usage. |
Farewell to Aimless Large-scale Pretraining: Influential Subset Selection for Language Model (2023.findings-acl)
Copied to clipboard
Xiao Wang, Weikang Zhou, Qi Zhang, Jie Zhou, SongYang Gao, Junzhe Wang, Menghan Zhang, Xiang Gao, Yun Wen Chen, Tao Gui
| Challenge: | Pretrained language models have achieved remarkable success in various natural language processing tasks. |
| Approach: | They propose to use end-task knowledge to select a tiny subset of pretraining corpus to influence performance. |
| Outcome: | The proposed model outperforms pretrained models on eight datasets covering four domains with 0.45% of the data and a three-orders-of-magnitude lower computational cost. |
Automated Cognate Detection as a Supervised Link Prediction Task with Cognate Transformer (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing methods for cognate identification are based on distributions of phonemes and make little use of cognacy labels. |
| Approach: | They propose a transformer-based architecture inspired by computational biology for automated cognate detection. |
| Outcome: | The proposed architecture performs better than existing methods with increased supervision. |
Fast and Accurate Deep Bidirectional Language Representations for Unsupervised Learning (2020.acl-main)
Copied to clipboard
| Challenge: | Existing deep bidirectional language models are limited by repetitive inferences on unsupervised tasks for the computation of contextual language representations. |
| Approach: | They propose a deep bidirectional language model called a Transformer-based Text Autoencoder (T-TA) it computes contextual language representations without repetition and shows competitive or even better accuracies than BERT . |
| Outcome: | The proposed model performs six times faster on a reranking task and twelve times faster in a semantic similarity task. |
On the Practical Computational Power of Finite Precision RNNs for Language Recognition (P18-2)
Copied to clipboard
| Challenge: | Recurrent Neural Networks (RNNs) are famously known to be Turing complete, but this relies on infinite precision in the states and unbounded computation time. |
| Approach: | They propose to use LSTM and Elman-RNN with ReLU activation to study RNNs . they show that LS and ReLU-RNns can easily implement counting behavior . |
| Outcome: | The LSTM and the Elman-RNN with ReLU activation are stronger than the RNN with squashing activation and the GRU. |
Great Power, Great Responsibility: Recommendations for Reducing Energy for Training Language Models (2022.findings-naacl)
Copied to clipboard
| Challenge: | Recent studies show that the energy requirements of current NLP models are growing at a rapid, unsustainable pace. |
| Approach: | They investigate ways to measure energy usage and different hardware settings that can be tuned to reduce energy consumption for training and inference for language models. |
| Outcome: | The proposed techniques can reduce energy consumption for training and inference for language models. |
ASETF: A Novel Method for Jailbreak Attack on LLMs through Translate Suffix Embeddings (2024.emnlp-main)
Copied to clipboard
| Challenge: | Attaching suffixes to harmful instructions can hack the defense of Large language models (LLMs) However, due to the unreadable of adversarial suffix, it can be relatively easily penetrated by common defense methods such as perplexity filters. |
| Approach: | They propose an algorithm to embed adversarial suffixes into coherent and understandable text to attack Large language models (LLMs) using a Advbench dataset. |
| Outcome: | The proposed approach reduces the computation time of adversarial suffixes and achieves a much better attack success rate than existing techniques. |
Personalized Pieces: Efficient Personalized Large Language Models through Collaborative Efforts (2024.emnlp-main)
Copied to clipboard
| Challenge: | Experimental results show Per-Pcs outperforms non-personalized and PEFT retrieval baselines, offering performance comparable to OPPU with significantly lower resource use across six tasks. |
| Approach: | They propose a framework that allows users to safely share and assemble personalized large language models using their history data. |
| Outcome: | Experimental results show that Per-Pcs outperforms non-personalized and PEFT retrieval baselines with significantly lower resource use across six tasks. |
HOTTER: Hierarchical Optimal Topic Transport with Explanatory Context Representations (2021.findings-emnlp)
Copied to clipboard
Sabine Wehnert, Christian Scheel, Simona Szakács-Behling, Maret Nieländer, Patrick Mielke, Ernesto William De Luca
| Challenge: | Natural language processing (NLP) is often the backbone of today’s systems for user interactions, information retrieval and others. |
| Approach: | They propose an extension to a specific emerging hybrid document distance metric which combines topic models and word embeddings. |
| Outcome: | The proposed method is competitive on public datasets and the language model BERT is used for a document categorization task. |
Hierarchical Trivia Fact Extraction from Wikipedia Articles (2020.coling-main)
Copied to clipboard
| Challenge: | Existing methods for extracting trivia facts for Wikipedia categories are not efficient . a trivia fact is an interesting fact that is unusual, unexpected, or unique . |
| Approach: | They propose an unsupervised algorithm that automatically mines trivia facts for a given entity . they propose to target at a single Wikipedia article and leverage its hierarchical structure . |
| Outcome: | The proposed algorithm outperforms existing methods and is 100 times faster than existing methods. |
AnchorAttention: Difference-Aware Sparse Attention with Stripe Granularity (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for large language models with extended context lengths face significant computational challenges during the prefill phase. |
| Approach: | They propose a difference-aware, dynamic sparse attention mechanism that efficiently identifies critical attention regions at a finer stripe granularity while adapting to global contextual information. |
| Outcome: | The proposed model achieves a speedup of 1.44 while maintaining higher recall rates. |
On the Representational Capacity of Recurrent Neural Language Models (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies have focused on LMs as formal languages, but they do not consider language membership. |
| Approach: | They extend the Turing completeness result to the probabilistic case . they show that a rationally weighted RLM can simulate any deterministic Turing machine . |
| Outcome: | The proposed model can simulate any deterministic Turing machine with rationally weighted transitions . the proposed model is based on recurrent neural networks with a rational weighting over strings . |
Direct Metric Optimization for Image Captioning through Reward-Weighted Augmented Data Utilization (2024.acl-long)
Copied to clipboard
Takumi Takada, Yuma Suzuki, Hiroki Takushima, Hayato Tanoue, Haruki Sato, Aiswariya Kumar, Hiroki Nishihara, Takayuki Hori, Kazuya Ueki
| Challenge: | Recent large-scale vision language models (VLMs) lack continuity between learning objective and performance metrics. |
| Approach: | They propose a lightweight final-metric-optimizing training method that replaces the expensive exploration process in RL with an offline, diverse text data augmentation method. |
| Outcome: | The proposed method achieves comparable performance to state-of-the-art RL method while saving hundreds of times more model forwarding iterations and greater amounts of computation time. |
Revisiting Character-Based Neural Machine Translation with Capacity and Compression (D18-1)
Copied to clipboard
| Challenge: | Translating characters instead of words or word-fragments can simplify the processing pipeline but results in longer sequences . |
| Approach: | They propose to use sequence-to-sequence architectures of sufficient depth to solve the problem . they also evaluate the performance versus computation time tradeoffs they offer . |
| Outcome: | The proposed models outperform models operating over word fragments in character-level NMT, the authors show . they also show that the proposed models do not match the performance of their deep character baseline model . |
Data Selection for Bilingual Lexicon Induction from Specialized Comparable Corpora (2020.coling-main)
Copied to clipboard
| Challenge: | Narrow specialized comparable corpora are small in size, making it difficult to build efficient models to acquire translation equivalents. |
| Approach: | They propose to use Tf-Idf and cross entropy to improve bilingual lexicon induction from specialized comparable corpora by a factor of 10 . |
| Outcome: | The proposed methods improve bilingual lexicon induction by a large margin. |
Powerformer: Efficient and High-Accuracy Privacy-Preserving Language Model with Homomorphic Encryption (2025.acl-long)
Copied to clipboard
| Challenge: | a new privacy-preserving language model, Powerformer, is designed to reduce computation overhead while maintaining model performance. |
| Approach: | They propose an efficient homomorphic encryption-based privacy-preserving language model . it incorporates three key techniques to optimize encrypted computations . |
| Outcome: | The proposed model achieves 45% reduction in computation time compared to state-of-the-art models . authors say the model preserves data privacy and AI capabilities in MLaaS environments . |
Cold-start Active Learning through Self-supervised Language Modeling (2020.emnlp-main)
Copied to clipboard
| Challenge: | Labeling data is a fundamental bottleneck in machine learning due to annotation cost and time. |
| Approach: | They propose a strategy that uses the pre-training loss to find examples that surprise the model and minimize labeling costs. |
| Outcome: | The proposed approach reduces labeling costs and costs by using pre-trained language models. |
Text Anomaly Detection with Simplified Isolation Kernel (2025.findings-emnlp)
Copied to clipboard
| Challenge: | High-dimensional dense embeddings extracted by large language models pose memory requirements and high computation time. |
| Approach: | They propose a method that maps high-dimensional dense embeddings to lower-dimensional sparse representations while preserving crucial anomaly characteristics. |
| Outcome: | The proposed method achieves better detection performance than 11 SOTA anomaly detection algorithms while maintaining computational efficiency and low memory cost. |
The Efficiency vs. Accuracy Trade-off: Optimizing RAG-Enhanced LLM Recommender Systems Using Multi-Head Early Exit (2025.acl-long)
Copied to clipboard
Huixue Zhou, Hengrui Gu, Zaifu Zhan, Xi Liu, Kaixiong Zhou, Yongkang Xiao, Mingfu Liang, Srinivas Prasad Govindan, Piyush Chawla, Jiyan Yang, Xiangfei Meng, Huayu Li, Buyun Zhang, Liang Luo, Wen-Yen Chen, Yiping Han, Bo Long, Rui Zhang, Tianlong Chen
| Challenge: | Existing frameworks for Large Language Models (LLMs) for Click-Through Rate prediction require a careful balance between computational efficiency and predictive accuracy. |
| Approach: | They propose a framework that integrates Retrieval-Augmented Generation with a novel multi-head early exit architecture to address both challenges. |
| Outcome: | The proposed framework reduces retrieval time while maintaining high model performance. |